Searching content using a dimensional database

ABSTRACT

A software facility for selecting documents is disclosed. The facility maintains a dimensional model of a group of documents. The dimensional model reflects values for a number of differentiated attributes for each of the documents of the group. The facility receives a query specifying values for one or more of these attributes. In response to receiving the query, the facility uses the dimensional model to generate a list of documents in the group having the attribute values specified by the query.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application is a divisional application of U.S. patent applicationSer. No. 11/751,465, entitled “Searching Documents Using a DimensionalDatabase,” filed Mar. 21, 2007, which is a continuation of U.S. patentapplication Ser. No. 09/954,719, entitled “Searching Documents Using aDimensional Database,” filed Sep. 17, 2001, now U.S. Pat. No. 7,493,308,issued Feb. 17, 2009, which claims the benefit of U.S. ProvisionalApplication No. 60/237,672, entitled “Multidimensional Search ofDocuments,” filed Oct. 3, 2000, which are all hereby incorporated hereinby references.

TECHNICAL FIELD

The present invention is directed toward the field of content searching,and more particularly to the field of web-based document searching.

BACKGROUND OF THE INVENTION

The World Wide Web (“the web”) is a distributed international electroniclibrary of documents and other data resources. A particular document isaccessed on the web using a unique identifier for the document, called a“URL,” short for Uniform Resource Locator. If a user seeking to access aparticular document has the URL for the document, s/he may simply typeit into the URL field of a web browser. In many cases, the URL for adocument may be obtained from a second, related document containing alink to the first document.

It is conservatively estimated that over a billion documents areavailable on the web. (Indeed, smaller “webs,” such as “Intranets” usedonly by the employees of a particular business, may themselves provideaccess to hundreds of thousands of documents.) For a particular userhaving a particular need, the web may contain several documents thataddress the need, all unknown to the user. For example, for a userinterested in details of the 1955 grape harvest in Eastern Washington,15 documents may be available on the web that contain such information,all unknown to the user.

In order to help users identify documents on the web relating toparticular subjects, hierarchical web directories and web search engineshave been developed. A hierarchical web directory is a set ofhuman-compiled lists of documents available via the web each relating toa particular subject represented in a hierarchy of topics. Table 1 belowshows a designation of a hierarchical web directory topic correspondingto a list of documents available via the web that includes documentscontaining information about the 1955 grape harvest in EasternWashington.

TABLE 1 Society and Culture  Food and Drink   Spirits    Wine    Regional      Eastern Washington

The topic corresponding to the list, Eastern Washington, is a subtopicof the topic “Regional,” which is a subtopic of the topic “Wine,” whichis a subtopic of the topic “Spirits,” which is a subtopic of the topic“Food and Drink,” which is a subtopic of the topic “Society andCulture.” In order to provide a hierarchical web directory, its providermust create a hierarchy of topics, identify documents available via theweb, and identify topics to whose lists the identified documents shouldbe added.

A web search engine, on the other hand, allows users to type one or morekey words and returns a list of documents containing those keywords. Inparticular, web search engines typically include documents in the listthat have the highest percentages of occurrences of the key words amongall of the documents. For example, to identify documents containingdetails of the 1955 grape harvest in Eastern Washington, a user mighttype the key word string “1955 grape harvest Eastern Washington.” Theweb search engine processes such queries against a database representingthe contents of as many web pages as possible, typically gathered by“spidering,” or automatically traversing links from known web pages tonew web pages.

Both of these conventional approaches to identifying documents on theweb have significant disadvantages. Hierarchical web directories areextremely labor intensive, requiring human editors to review andcategorize web documents. This reliance on manual processes oftenresults in outdated or inaccurate content. Also, hierarchical webdirectories are only usable to identify web pages relating to topicscreated by human editors. Hierarchical web directories are alsodifficult for users to successfully use, as a user must typically selectthe exact same sequence of subtopics as the person that catalogued theweb site.

Web search engines, while not typically suffering from the deficienciesof hierarchical web directories relating to their manual nature, havethe disadvantage that they rely on the occurrence of particular keywords in sought web pages. Because many words have multiple meanings,Web search engines often generate false positive matches, where akeyword appears in the Web page in a different sense than the senseintended by the user in formulating the query. On the other hand,because of the large number of words that can be used to get across thesame idea, Web search engines also often generate false negativematches. Web search engines also typically filter out noise words thatoccur in most web pages, such as “an” or “if,” which make it impossibleto search for web pages using these words when using a web searchengine. Further, aside from applying certain frequency analysistechniques, web search engines typically ignore the specific usage andsignificance of particular key words in the searched web pages.

Accordingly, a more effective approach to identifying documents on theweb and in other electronic libraries would have significant utility.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a high-level block diagram of the computer system upon whichthe facility preferably executes.

FIG. 2 is a flow diagram showing the steps preferably performed by thefacility in order to construct a dimensional model.

FIG. 3 is a diagram showing a registration form completed by thewebmaster of a company root page.

FIG. 4 is a diagram showing the source code for a sample company rootpage as automatically located and parsed by the facility.

FIG. 5 is the data structure diagram showing the definition of a sampledimensional model in dimensional database notation.

FIG. 6 is a data structure diagram showing the data tables thatpreferably underlie the sample dimensional model shown in FIG. 5.

FIG. 7 is a flow diagram showing the steps preferably performed by thefacility in order to process search requests against the dimensionalmodel.

FIG. 8 is a diagram showing a sample search request.

FIG. 9 is a diagram showing a representation of the search request shownin FIG. 8 expressed in a search request star notation.

FIG. 10 is a diagram showing a second sample search request for the samedimensional model.

FIG. 11 is a diagram showing the second sample search request expressedin search request star notation.

FIG. 12 is a diagram showing a search request, expressed in starnotation, for a yellow pages entry, i.e., an entry in a businesstelephone directory.

FIG. 13 is a diagram showing a search request, expressed in searchrequest star notation, for descriptions of products sold by a computersuperstore.

FIG. 14 is a diagram showing a search request, expressed in searchrequest star notation, for a library resource, such as a book or anaudio or video recording.

DETAILED DESCRIPTION

Embodiments of the present invention provide a software facility forselecting documents or other types of resources using a dimensionalmodel (“the facility”). In some embodiments, the facility is adapted toselect web pages in response to user queries.

The facility preferably generates and maintains a dimensional model of anumber of documents, such as web pages. For each modeled document, themodel contains information about the document in at least a portion of anumber of different informational dimensions. For example, for documentsthat are home pages of companies, the facility preferably includes inthe model for each modeled document indications of the name, type,category, and location of the company. The facility preferably obtainsthis information using one or more of a variety of techniques, includingmanual or automatic submission, or automatic spidering combined withparsing and/or natural language understanding.

The facility preferably maintains the information obtained for modeleddocuments in a dimensional model. The dimensional database techniquesdiscussed herein are well-known to those skilled in the art, and aredescribed in greater detail in Kimball, Ralph: “The Data WarehouseToolkit Practical Techniques for Building Dimensional Data Warehouses,”1996. The model includes a fact table that contains a row for each ofthe modeled documents. For example, the fact table may be comprised ofrows each corresponding to a company home page, and containing the URLfor that company home page. The model further includes a dimension tablefor each of the modeled dimensions. For example, the model may includefour dimension tables, one each for the company_name, company_type,company_category, and company_location dimensions. Each dimension tablehas a row for each unique value of the dimension. For example, thecompany_location dimension table may have a row for each unique locationof one or more modeled company root pages.

The model may be normalized by joining all of the rows of all of thedimension tables to the fact table, generating a result table in whicheach row explicitly contains all of the information about one document.This result table, however, can be many times larger than the entiremodel, and is less efficient than the model for use in executingqueries.

The facility receives queries that specify functions on at least some ofthe modeled dimensions. For example, the facility may receive a querythat specifies values for three of the four modeled dimensions. Toprocess the query, for each specified dimension value, the facilityselects the rows of the corresponding dimension table that match thevalue. The facility then joins the selected rows in the dimension tablesto the fact table. The result table from this join is generally ofmanageable size, and contains a row for each document that satisfies thequery, and is preferably used by the facility to generate a query resultthat the user can use to access the documents that satisfies the query.

In this manner, document queries that more effectively describe soughtdocuments may be processed without requiring significant manual efforton the part of the operators of the facility or relying on the adequacyof such effort, producing more useful query results than conventionaldocument searching approaches.

FIG. 1 is a high-level block diagram of the computer system upon whichthe facility preferably executes. The computer system 100 contains oneor more central processing units (CPUs) 110, input/output devices 120,and a computer memory (memory) 130. Among the input/output devices is apersistent storage device 121, such as a hard disk drive, and acomputer-readable media drive 122, which can be used to install softwareproducts, including components of the facility, which are provided on acomputer-readable medium, such as a CD-ROM. The input/output devicesalso include a network connection 123, through which the computer system100 may be connected to the network to be analyzed by the facility. Thememory 130 preferably contains the dimensional search facility 131, aswell as dimensional models of document sets 132 and 133, both preferablygenerated and used by the facility. While items 131-133 are preferablystored in memory while being used, those skilled in the art willappreciate that these items, or portions of them, may be transferredbetween memory and the persistent storage device for purposes of memorymanagement and data integrity. While the facility is preferablyimplemented on a computer system configured as described above, thoseskilled in the art will recognize that it may also be implemented oncomputer systems having different configurations, or distributed acrossmultiple computer systems.

FIG. 2 is a flow diagram showing the steps preferably performed by thefacility in order to construct a dimensional model. Those skilled in theart will appreciate that these steps could be used to construct severaldifferent dimensional models simultaneously. In step 201, the facilityobtains a registration record containing the URL for a company root pageor other modeled document, as well as dimension values describing thecompany, or other attributes of the document. The facility preferablyobtains this registration information in a variety of different ways,including receiving a registration form filled out by someoneresponsible for the document, such as the webmaster for a company rootpage, or such as by automatically exploring and analyzing documents.

FIG. 3 is a diagram showing a registration form completed by thewebmaster of a company root page. The form 310 contains a field 320 inwhich the URL used to identify the company root page has been entered.It will be appreciated by those skilled in the art that the facility mayuse many different types of references, or “links,” to refer to modeleddocuments and other resources. For example, URLs specified in accordancewith RFC 1738 may be used to refer to online resources, as may variousother electronic naming conventions. Additionally, various schemes, suchas street addresses or Dewey decimal numbers, may be used to refer tooffline resources. Fields 321-324 are fields into which attribute valuesfor the company root page have been entered.

FIG. 4 is a diagram showing the source code for a sample company rootpage as automatically located and parsed by the facility. Source codesegment 400 shows the first eleven lines of the source code. As thesource code was retrieved by the facility using the URL for the companyroot page, the URL for the source code is known to the facility.Attribute values for the company root page are specified on lines 3-6 inthe XML block occurring in lines 2-7. These lines are preferablyinserted into the HTML source code by the webmaster of the company rootpage to support indexing by the facility, as well as for a variety ofother purposes. Embedding XML source code within HTML source code inthis manner has been proposed by some commentators to support theinclusion of XML data in existing HTML documents. Alternatively,attribute information may be included in existing HTML constructs, suchas the HTML <span> tag. For example, the attribute value informationshown on line 3 would be encoded using the <span> tag as follows:

“<span class=company_name> Hughes Satellite, Inc. </span>”

Where the company root page is expressed in XML, the attributeinformation is merely included in custom attribute tags that are addedby the webmaster to the XML document comprising the root page.

Returning to FIG. 2, in step 202, the facility adds a row to the facttable of the dimensional model for the registration record obtained instep 201. The facility further adds rows to dimension tables of thedimensional model as necessary. After step 202, the facility continuesin step 201 to obtain the next registration record.

FIG. 5 is the data structure diagram showing the definition of a sampledimensional model in dimensional database notation. The model has asingle fact table 510, and four dimension tables 520, 530, 540, and 550.Dimension table 520 is for the company_name dimension and contains acompany_name field 522 and a company_name_key field 521 in each row.Dimension table 530 is for the company_dimension, and contains acompany_type field 532 and a company_type_key field in each row.Dimension table 540 is for the company_location dimension, and containsa company_location field 542 and a company_location_key field 541 ineach row. Dimension table 550 is for the company_category dimension, andcontains a company_category field 552 and a company_category_key field551 in each row. The fact table 510 contains a row for each modeleddocument. Each row contains a company_name_key field 511, which containsthe key corresponding in the company_name dimension table 520 the valueof the company_name attribute for the company root page to which the rowof the fact table corresponds. The line between fields 511 and 521indicates that tables 510 and 520 can be joined on the company_name_keyfield. Such a join operation, if applied to all of the rows of bothtables, would produce a result table containing all of the informationin the fact table, plus an additional column containing the company_namefield for each row. The fact table further contains key fields 512, 513,and 514, through which the fact table may be joined to dimension tables530, 540, and 550, respectively. Finally, fact table 510 includes acompany_root_page_link field 515 that, in each row, contains a link tothe company root page to which the row corresponds. In one embodiment,in which two or more documents may have all the same attribute values, arow of the fact table may correspond to more than one document. In thiscase, rows of the fact table that correspond to more than one modeleddocument preferably include the URL to a web page containing a list ofthese documents rather than a URL directly to a particular document. Inan additional embodiment, rows that contain such a URL further includean indication of the number of documents listed at that URL. Thisindication may be a count of these documents, may indicate a range intowhich the number of documents falls, or may simply indicate whether thenumber of listed documents makes the list of documents too large toretrieve, at least under certain circumstances.

In some embodiments, the facility supports hierarchical dimensions. Forexample, rather than merely reflecting a city, a more thoroughhierarchical notion of the company_location dimension as shown below inTable 2 may be used:

TABLE 2 Country  State   City

Such hierarchical dimensions may be represented either in a singledimension table containing a column for each hierarchical component ofthe dimension, i.e., a single company_location dimension tablecontaining separate columns for country, state, and city, or by using a“snowflaked” sequence of dimension tables emanating from the fact tablein decreasing level of detail, i.e., a dimension table for city,referred to by the fact table and referring to a dimension table forstate, which in turn refers to a dimension table for country.

FIG. 6 is a data structure diagram showing the data tables thatpreferably underlie the sample dimensional model shown in FIG. 5. Thediagram shows fact table 610, as well as dimension tables 620, 630, 640,and 650. The fact table 610 has columns 611-614, each corresponding tothe key of one of the dimension tables. For example, sample row 616,corresponding to the sample company root page whose attribute values areshown in FIGS. 3 and 4, contains the company_name_key 237, as does row626 of dimension table 620 for the company_name dimension. That row ofdimension table 620 indicates that the company root page to which row616 of the fact table corresponds has the value “Hughes Satellite, Inc.”for the company_name attribute. The fact table 610 further includescolumn 615, which contains in each row the URL of the company root pageto which the row corresponds, in this case,“http://www.hughessatellite.com”. When the facility obtains theattribute information for this company root page, it adds row 616 to thefact table 610. It further searches each of the dimension tables todetermine whether the obtained attribute value for that dimension isalready contained in the dimension table. If so, the facility merelycopies the key for that row of the dimension table into the row beingadded to the fact table. If the attribute value is not already in thedimension table, then the facility adds a row to the dimension tablecontaining the attribute value and a key that is unique within thedimension table, and adds that key to the row being added to the facttable.

FIG. 7 is a flow diagram showing the steps preferably performed by thefacility in order to process search requests against the dimensionalmodel. In step 701, the facility receives a search request thatspecifies dimension values for one or more selected dimensions. Ratherthan specifying a value for each such dimension, in some embodiments,the search request may specify other tests with respect to a dimension,such as whether the dimension value is non-null, whether it falls withina particular range of dimension values or within a list of dimensionvalues, whether it matches a pattern, or whether it satisfies anarbitrary programmatic or mathematical function.

FIG. 8 is a diagram showing a sample search request. The search request810 contains dimension values 822-824 provided by a user for thecompany_type, company_location, and company_category dimensions,respectively. This search request is for the company root pages ofservice provider companies in the satellite communications area that arelocated in California.

FIG. 9 is a diagram showing a representation of the search request shownin FIG. 8 expressed in a search request star notation. The searchrequest star shown is comprised of nodes 910, 920, 930, 940, and 950.The search request star shown specifies the value “service provider” forthe company_type dimension in node 930, specifies the value “California”for the company_location dimension in node 940, and specifies the value“satellite communications” for the company_category dimension in node950.

FIG. 10 is a diagram showing a second sample search request for the samedimensional model. The diagram shows a search request for the companyroot pages of companies whose names contain the word “if”. FIG. 11 is adiagram showing the same search request expressed in search request starnotation. Search requests such as this one may be executed using avariety of well-known searching techniques. These include regularexpression matching techniques that may be coded and called from withina database server, such as an Oracle database server, as well astechniques using an SQL-like predicate. Those skilled in the art willrecognize that additional techniques may also be employed.

Returning to FIG. 7, the facility repeats steps 702-704 for eachdimension selected in the search request. In step 703, the facilitysubsets the dimension table for the selected dimension down to the rowsmatching the dimension value specified in the search request for theselected dimension or satisfying the tests specified in the searchrequest for the selected dimension. In other words, the facility selectsthe rows of the dimension table for the selected dimension whosedimension values match the specified dimension value. In step 704, ifadditional selected dimensions remain to be processed, the facilitycontinues in step 702 to process the next selected dimension. When allof the selected dimensions have been processed, the facility continuesin step 705. In step 705, the facility joins the dimension tables forthe selected dimensions, as subsetted in step 703, to the fact table ofthe dimensional model. The result table produced by this join operationcontains a row for each modeled document that satisfies the searchrequest. Each row of the result table preferably contains a URL or otherreference to the document satisfying the search, and, optionally, thevalues of attributes of each document, including attributescorresponding to dimensions not selected in the search request. Forexample, when each document is a book, the result table may include thecurrent rank of the book on the New York Times bestseller list. In someembodiments, the dimension processing represented by steps 702-704 andthe fact processing represented in step 705 are parallelized and proceedconcurrently (not shown). In step 706, the facility displays a searchresult to the user based on the table produced by the join operation.After step 706, the facility continues in step 701 to process the nextsearch request.

As will be appreciated by those skilled in the art, the facility may beemployed to generate and respond to search requests for dimensionalmodels for a wide variety of document groups and the attributes of theconstituent documents. FIGS. 12-14 show a few examples for other groupsof documents.

FIG. 12 is a diagram showing a search request, expressed in searchrequest star notation, for yellow pages entries, i.e., entries in abusiness telephone directory. As can be seen from nodes 1220, 1230,1240, and 1250, the model for yellow pages entries models the entries onthe following attributes: vendor name, vendor location, product_type,and payment_types. As can be seen from nodes 1220, 1230, and 1240, thequery shown is for vendors whose names contain the word “Roma”, whoselocations contain the word “Colorado”, and whose product types includethe word “pizza”.

FIG. 13 is a diagram showing a search request, expressed in searchrequest star notation, for descriptions of products sold by a computersuperstore. As can be seen from nodes 1320, 1330, 1340, 1350, and 1360,the model for computer superstore product descriptions models thedescriptions on the following attributes: vendor, product_name,product_type, operating_system, and price. As can be seen from nodes1320 and 1340, the query shown is for computer superstore productdescriptions for microphone products from Dragon Systems.

FIG. 14 is a diagram showing a search request, expressed in searchrequest star notation, for a library resource, such as a book or anaudio or video recording. As can be seen from nodes 1420, 1430, 1440,1450, and 1460, the model for library resources models library resourceson the following attributes: title, author, publisher, subject, andpublication_date. As can be seen from nodes 1430 and 1460, the queryshown is for library resources published in 1998 by an author namedHenley.

It will be understood by those skilled in the art that theabove-described facility could be adapted or extended in various ways.For example, the facility may be used to model and select documents,other data artifacts, or other resources of virtually any type,including programmatic objects, such as COM or CORBA components or Javaapplets. While the foregoing description makes reference to preferredembodiments, the scope of the invention is defined solely by the claimsthat follow and the elements recited therein.

1. A computer-implemented method for collecting information about agroup of documents, comprising: under control of one or more computersystems configured with executable instructions, seeding the group ofdocuments with one or more seed documents; identifying, in at least aportion of the group of documents, at least one reference to at leastone secondary document; adding to the group of documents at least onesecondary document obtainable using the identified at least onereference; identifying, for at least a portion of the documents in thegroup of documents, two or more attribute values associated with atleast one aspect of the document; adding at least a portion of theidentified attribute values to a set of information associated with thegroup of documents, the set of information enabling documents in thegroup of documents to be located based at least in part upon any of theattribute values in the set of information; applying a test to at leastone secondary document; and adding the attribute values identified forthe at least one secondary document to the set of information associatedwith the group of documents only if the applied test is satisfied. 2.The computer-implemented method of claim 1, wherein at least onereference comprises a hyperlink.
 3. The computer-implemented method ofclaim 1, wherein at least a portion of the secondary documents added tothe group of documents comprise web pages.
 4. The computer-implementedmethod of claim 1, further comprising, for each secondary document:applying a test to the secondary document; and adding the secondarydocument to the group of documents only if the applied test issatisfied.
 5. The computer-implemented method of claim 1, furthercomprising: receiving a search request specifying attribute values foreach of one or more predetermined document attributes and requesting areport of documents from the group of documents having the specifiedattribute values for the one or more predetermined document attributes.6. The computer-implemented method of claim 5, further comprising:processing the received search request using the group of documents andthe associated set of information.
 7. The computer-implemented method ofclaim 6, wherein processing the received search request furthercomprises: determining documents in the group of documents having thespecified attribute values for the one or more predetermined documentattributes based at least in part upon the associated set ofinformation; and transmitting a report of the determined documents to asource of the request.
 8. The computer-implemented method of claim 1,wherein the attribute values are obtained for at least a portion of thegroup of documents by traversing references between documents of thegroup and parsing the contents of traversed-to documents.
 9. Thecomputer-implemented method of claim 1, wherein attribute values areobtained for at least a portion of the plurality of documents byreceiving submissions indicating the attribute values associated witheach document.
 10. A system for collecting information about a group ofdocuments, comprising: at least one processor; and memory includinginstructions that, when executed by the at least one processor, causethe system to: seed the group of documents with one or more seeddocuments; identify, in at least one seed document, at least onereference to at least one secondary document; if a first test issatisfied, add to the group of documents at least one secondary documentobtainable using the identified at least one reference; identify, for atleast a portion of the documents in the group of documents, two or moreattribute values associated with at least one aspect of the document;and if a second test is satisfied, add at least a portion of theidentified attribute values to a set of information associated with thegroup of documents, the set of information enabling documents in thegroup of documents to be located based at least in part upon any of theattribute values in the set of information.
 11. The system of claim 10,wherein at least one reference comprises a hyperlink, and at least aportion of the secondary documents added to the group comprise webpages.
 12. The system of claim 10, wherein the instructions whenexecuted further cause the system to: receive a search requestspecifying attribute values for each of one or more predetermineddocument attributes and request a report of documents from the group ofdocuments having the specified attribute values for the one or morepredetermined document attributes based at least in part upon theassociated set of information.
 13. The system of claim 12, wherein thesearch request is received from a source, the instructions when executedfurther causing the system to: determine documents in the group ofdocuments having the specified attribute values for the one or morepredetermined document attributes based at least in part upon theassociated set of information; and transmit a report of the determineddocuments to the source.
 14. A computer readable storage mediumincluding instructions that, when executed by a processor, cause theprocessor to: seed a group of documents with one or more seed documents;and identify, in at least one seed document, at least one reference toat least one secondary document; if a first test is satisfied, add tothe group of documents at least one secondary document obtainable usingthe identified at least one reference; identify, for at least a portionof the documents in the group of documents, two or more attribute valuesassociated with at least one aspect of the document; and if a secondtest is satisfied, add at least a portion of the identified attributevalues to a set of information associated with the group of documents,the set of information enabling documents in the group of documents tobe located based at least in part upon any of the attribute values inthe set of information.
 15. The computer readable storage medium ofclaim 14, wherein at least one reference comprises a hyperlink, and atleast a portion of the secondary documents added to the group compriseweb pages.
 16. The computer readable storage medium of claim 14, whereinthe instructions when executed further cause the system to: receive asearch request specifying attribute values for each of one or morepredetermined document attributes and request a report of documents fromthe group of documents having the specified attribute values for the oneor more predetermined document attributes based at least in part uponthe associated set of information.
 17. The computer readable storagemedium of claim 16, wherein the search request is received from asource, the instructions when executed further causing the system to:determine documents in the group of documents having the specifiedattribute values for the one or more predetermined document attributesbased at least in part upon the associated set of information; andtransmit a report of the determined documents to the source.